Failure Archaeology and Anomaly Detection for Mid-Sized Internet Sites

Mid-sized Internet sites have enough complexity to be failure prone, but do not have enough resources to create the resilient architectures used by large Internet sites. In this paper, we characterize failures seen in mid-sized Internet sites and evaluate the use of (i) visualization techniques to assist humans in pinpointing failures and (ii) statistical learning techniques for automated failure prediction, detection, and localization. Analysis of failure reports and HTTP request logs shows that anomalies experienced by mid-sized sites are often caused by single page bugs that can affect many other pages on the site. These anomalies can be classified as short-term outages or volume traffic anomalies. We evaluate the statistical learning techniques of Principal Component Analysis and Naive Bayes. PCA is best at detecting short-term anomalies and the wide variations in traffic that often precede them, as well as localizing failures to specific pages. A well-trained Naive Bayes algorithm can detect longerterm volume anomalies. When applying these techniques to realworld web request traffic, all of the human-detected failures are labeled as anomalous. Our techniques also detected other traffic irregularities that had not been explained to us by the administrators, suggesting that these techniques would be useful in the future.